Shared buffered memory routing

ABSTRACT

A shared memory controller receives a flit from another first shared memory controller over a shared memory link, where the flit includes a node identifier (ID) field and an address of a particular line of the shared memory. The node ID field identifies that the first shared memory controller corresponds to a source of the flit. Further, a second shared memory controller is determined from at least the address field of the flit, where the second shared memory controller is connected to a memory element corresponding to the particular line. The flit is forwarded to the second shared memory controller using a shared memory link according to a routing path.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/665,541, filed on Aug. 1, 2017, which claims the benefit of U.S.application Ser. No. 14/670,578, filed on Mar. 27, 2015, and Issued asU.S. Pat. No. 9,720,838, on Aug. 1, 2017, the entire contents of whichare hereby incorporated by reference.

FIELD

This disclosure pertains to computing system, and in particular (but notexclusively) to memory access between components in a computing system.

BACKGROUND

Advances in semi-conductor processing and logic design have permitted anincrease in the amount of logic that may be present on integratedcircuit devices. As a corollary, computer system configurations haveevolved from a single or multiple integrated circuits in a system tomultiple cores, multiple hardware threads, and multiple logicalprocessors present on individual integrated circuits, as well as otherinterfaces integrated within such processors. A processor or integratedcircuit typically comprises a single physical processor die, where theprocessor die may include any number of cores, hardware threads, logicalprocessors, interfaces, memory, controller hubs, etc.

As a result of the greater ability to fit more processing power insmaller packages, smaller computing devices have increased inpopularity. Smartphones, tablets, ultrathin notebooks, and other userequipment have grown exponentially. However, these smaller devices arereliant on servers both for data storage and complex processing thatexceeds the form factor. Consequently, the demand in thehigh-performance computing market (i.e. server space) has alsoincreased. For instance, in modern servers, there is typically not onlya single processor with multiple cores, but also multiple physicalprocessors (also referred to as multiple sockets) to increase thecomputing power. But as the processing power grows along with the numberof devices in a computing system, the communication between sockets andother devices becomes more critical.

In fact, interconnects have grown from more traditional multi-drop busesthat primarily handled electrical communications to full blowninterconnect architectures that facilitate fast communication.Unfortunately, as the demand for future processors to consume at evenhigher-rates corresponding demand is placed on the capabilities ofexisting interconnect architectures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a computing system including aninterconnect architecture.

FIG. 2 illustrates an embodiment of a interconnect architectureincluding a layered stack.

FIG. 3 illustrates an embodiment of a request or packet to be generatedor received within an interconnect architecture.

FIG. 4 illustrates an embodiment of a transmitter and receiver pair foran interconnect architecture.

FIG. 5 illustrates an embodiment of a layered protocol stack associatedwith a high performance general purpose input/output (GPIO)interconnect.

FIG. 6 illustrates a representation of an example multi-slot flit.

FIG. 7 illustrates an example system utilizing buffered memory access.

FIG. 8A illustrates a simplified block diagram of an embodiment of anexample node.

FIG. 8B illustrates a simplified block diagram of an embodiment of anexample system including a plurality of nodes.

FIG. 8C illustrates another simplified block diagram of an embodiment ofan example system including a plurality of nodes.

FIG. 9 is a representation of data transmitted according to an exampleshared memory link.

FIG. 10A is a representation of data transmitted according to anotherexample of a shared memory link.

FIG. 10B is a representation of an example start of data framing token.

FIG. 11 is a representation of data transmitted according to anotherexample of a shared memory link.

FIG. 12 illustrates a representation of an example multi-slot flitmodified for routing within a shared memory architecture.

FIGS. 13A-13B are flowcharts illustrating example techniques for routingtransactions within a shared memory architecture.

FIG. 14 illustrates an embodiment of a block diagram for a computingsystem including a multicore processor.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth,such as examples of specific types of processors and systemconfigurations, specific hardware structures, specific architectural andmicro architectural details, specific register configurations, specificinstruction types, specific system components, specificmeasurements/heights, specific processor pipeline stages and operationetc. in order to provide a thorough understanding of the presentinvention. It will be apparent, however, to one skilled in the art thatthese specific details need not be employed to practice the presentinvention. In other instances, well known components or methods, such asspecific and alternative processor architectures, specific logiccircuits/code for described algorithms, specific firmware code, specificinterconnect operation, specific logic configurations, specificmanufacturing techniques and materials, specific compilerimplementations, specific expression of algorithms in code, specificpower down and gating techniques/logic and other specific operationaldetails of computer system haven't been described in detail in order toavoid unnecessarily obscuring the present invention.

Although the following embodiments may be described with reference toenergy conservation and energy efficiency in specific integratedcircuits, such as in computing platforms or microprocessors, otherembodiments are applicable to other types of integrated circuits andlogic devices. Similar techniques and teachings of embodiments describedherein may be applied to other types of circuits or semiconductordevices that may also benefit from better energy efficiency and energyconservation. For example, the disclosed embodiments are not limited todesktop computer systems or Ultrabooks™. And may be also used in otherdevices, such as handheld devices, tablets, other thin notebooks,systems on a chip (SOC) devices, and embedded applications. Someexamples of handheld devices include cellular phones, Internet protocoldevices, digital cameras, personal digital assistants (PDAs), andhandheld PCs. Embedded applications typically include a microcontroller,a digital signal processor (DSP), a system on a chip, network computers(NetPC), set-top boxes, network hubs, wide area network (WAN) switches,or any other system that can perform the functions and operations taughtbelow. Moreover, the apparatus', methods, and systems described hereinare not limited to physical computing devices, but may also relate tosoftware optimizations for energy conservation and efficiency. As willbecome readily apparent in the description below, the embodiments ofmethods, apparatus', and systems described herein (whether in referenceto hardware, firmware, software, or a combination thereof) are vital toa ‘green technology’ future balanced with performance considerations.

As computing systems are advancing, the components therein are becomingmore complex. As a result, the interconnect architecture to couple andcommunicate between the components is also increasing in complexity toensure bandwidth requirements are met for optimal component operation.Furthermore, different market segments demand different aspects ofinterconnect architectures to suit the market's needs. For example,servers require higher performance, while the mobile ecosystem issometimes able to sacrifice overall performance for power savings. Yet,it's a singular purpose of most fabrics to provide highest possibleperformance with maximum power saving. Below, a number of interconnectsare discussed, which would potentially benefit from aspects of theinvention described herein.

One interconnect fabric architecture includes the Peripheral ComponentInterconnect (PCI) Express (PCIe) architecture. A primary goal of PCIeis to enable components and devices from different vendors tointer-operate in an open architecture, spanning multiple marketsegments; Clients (Desktops and Mobile), Servers (Standard andEnterprise), and Embedded and Communication devices. PCI Express is ahigh performance, general purpose I/O interconnect defined for a widevariety of future computing and communication platforms. Some PCIattributes, such as its usage model, load-store architecture, andsoftware interfaces, have been maintained through its revisions, whereasprevious parallel bus implementations have been replaced by a highlyscalable, fully serial interface. The more recent versions of PCIExpress take advantage of advances in point-to-point interconnects,Switch-based technology, and packetized protocol to deliver new levelsof performance and features. Power Management, Quality Of Service (QoS),Hot-Plug/Hot-Swap support, Data Integrity, and Error Handling are amongsome of the advanced features supported by PCI Express.

Referring to FIG. 1, an embodiment of a fabric composed ofpoint-to-point Links that interconnect a set of components isillustrated. System 100 includes processor 105 and system memory 110coupled to controller hub 115. Processor 105 includes any processingelement, such as a microprocessor, a host processor, an embeddedprocessor, a co-processor, or other processor. Processor 105 is coupledto controller hub 115 through front-side bus (FSB) 106. In oneembodiment, FSB 106 is a serial point-to-point interconnect as describedbelow. In another embodiment, link 106 includes a serial, differentialinterconnect architecture that is compliant with different interconnectstandard.

System memory 110 includes any memory device, such as random accessmemory (RAM), non-volatile (NV) memory, or other memory accessible bydevices in system 100. System memory 110 is coupled to controller hub115 through memory interface 116. Examples of a memory interface includea double-data rate (DDR) memory interface, a dual-channel DDR memoryinterface, and a dynamic RAM (DRAM) memory interface.

In one embodiment, controller hub 115 is a root hub, root complex, orroot controller in a Peripheral Component Interconnect Express (PCIe orPCIE) interconnection hierarchy. Examples of controller hub 115 includea chipset, a memory controller hub (MCH), a northbridge, an interconnectcontroller hub (ICH) a southbridge, and a root controller/hub. Often theterm chipset refers to two physically separate controller hubs, i.e. amemory controller hub (MCH) coupled to an interconnect controller hub(ICH). Note that current systems often include the MCH integrated withprocessor 105, while controller 115 is to communicate with I/O devices,in a similar manner as described below. In some embodiments,peer-to-peer routing is optionally supported through root complex 115.

Here, controller hub 115 is coupled to switch/bridge 120 through seriallink 119. Input/output modules 117 and 121, which may also be referredto as interfaces/ports 117 and 121, include/implement a layered protocolstack to provide communication between controller hub 115 and switch120. In one embodiment, multiple devices are capable of being coupled toswitch 120.

Switch/bridge 120 routes packets/messages from device 125 upstream, i.e.up a hierarchy towards a root complex, to controller hub 115 anddownstream, i.e. down a hierarchy away from a root controller, fromprocessor 105 or system memory 110 to device 125. Switch 120, in oneembodiment, is referred to as a logical assembly of multiple virtualPCI-to-PCI bridge devices. Device 125 includes any internal or externaldevice or component to be coupled to an electronic system, such as anI/O device, a Network Interface Controller (NIC), an add-in card, anaudio processor, a network processor, a hard-drive, a storage device, aCD/DVD ROM, a monitor, a printer, a mouse, a keyboard, a router, aportable storage device, a Firewire device, a Universal Serial Bus (USB)device, a scanner, and other input/output devices. Often in the PCIevernacular, such as device, is referred to as an endpoint. Although notspecifically shown, device 125 may include a PCIe to PCI/PCI-X bridge tosupport legacy or other version PCI devices. Endpoint devices in PCIeare often classified as legacy, PCIe, or root complex integratedendpoints.

Graphics accelerator 130 is also coupled to controller hub 115 throughserial link 132. In one embodiment, graphics accelerator 130 is coupledto an MCH, which is coupled to an ICH. Switch 120, and accordingly I/Odevice 125, is then coupled to the ICH. I/O modules 131 and 118 are alsoto implement a layered protocol stack to communicate between graphicsaccelerator 130 and controller hub 115. Similar to the MCH discussionabove, a graphics controller or the graphics accelerator 130 itself maybe integrated in processor 105.

Turning to FIG. 2 an embodiment of a layered protocol stack isillustrated. Layered protocol stack 200 includes any form of a layeredcommunication stack, such as a Quick Path Interconnect (QPI) stack, aPCie stack, a next generation high performance computing interconnectstack, or other layered stack. Although the discussion immediately belowin reference to FIGS. 1-4 are in relation to a PCIe stack, the sameconcepts may be applied to other interconnect stacks. In one embodiment,protocol stack 200 is a PCIe protocol stack including transaction layer205, link layer 210, and physical layer 220. An interface, such asinterfaces 117, 118, 121, 122, 126, and 131 in FIG. 1, may berepresented as communication protocol stack 200. Representation as acommunication protocol stack may also be referred to as a module orinterface implementing/including a protocol stack.

PCI Express uses packets to communicate information between components.Packets are formed in the Transaction Layer 205 and Data Link Layer 210to carry the information from the transmitting component to thereceiving component. As the transmitted packets flow through the otherlayers, they are extended with additional information necessary tohandle packets at those layers. At the receiving side the reverseprocess occurs and packets get transformed from their Physical Layer 220representation to the Data Link Layer 210 representation and finally(for Transaction Layer Packets) to the form that can be processed by theTransaction Layer 205 of the receiving device.

Transaction Layer

In one embodiment, transaction layer 205 is to provide an interfacebetween a device's processing core and the interconnect architecture,such as data link layer 210 and physical layer 220. In this regard, aprimary responsibility of the transaction layer 205 is the assembly anddisassembly of packets (i.e., transaction layer packets, or TLPs). Thetransaction layer 205 typically manages credit-base flow control forTLPs. PCIe implements split transactions, i.e. transactions with requestand response separated by time, allowing a link to carry other trafficwhile the target device gathers data for the response.

In addition PCIe utilizes credit-based flow control. In this scheme, adevice advertises an initial amount of credit for each of the receivebuffers in Transaction Layer 205. An external device at the opposite endof the link, such as controller hub 115 in FIG. 1, counts the number ofcredits consumed by each TLP. A transaction may be transmitted if thetransaction does not exceed a credit limit. Upon receiving a response anamount of credit is restored. An advantage of a credit scheme is thatthe latency of credit return does not affect performance, provided thatthe credit limit is not encountered.

In one embodiment, four transaction address spaces include aconfiguration address space, a memory address space, an input/outputaddress space, and a message address space. Memory space transactionsinclude one or more of read requests and write requests to transfer datato/from a memory-mapped location. In one embodiment, memory spacetransactions are capable of using two different address formats, e.g., ashort address format, such as a 32-bit address, or a long addressformat, such as 64-bit address. Configuration space transactions areused to access configuration space of the PCIe devices. Transactions tothe configuration space include read requests and write requests.Message space transactions (or, simply messages) are defined to supportin-band communication between PCIe agents.

Therefore, in one embodiment, transaction layer 205 assembles packetheader/payload 206. Format for current packet headers/payloads may befound in the PCIe specification at the PCIe specification website.

Quickly referring to FIG. 3, an embodiment of a PCIe transactiondescriptor is illustrated. In one embodiment, transaction descriptor 300is a mechanism for carrying transaction information. In this regard,transaction descriptor 300 supports identification of transactions in asystem. Other potential uses include tracking modifications of defaulttransaction ordering and association of transaction with channels.

Transaction descriptor 300 includes global identifier field 302,attributes field 304 and channel identifier field 306. In theillustrated example, global identifier field 302 is depicted comprisinglocal transaction identifier field 308 and source identifier field 310.In one embodiment, global transaction identifier 302 is unique for alloutstanding requests.

According to one implementation, local transaction identifier field 308is a field generated by a requesting agent, and it is unique for alloutstanding requests that require a completion for that requestingagent. Furthermore, in this example, source identifier 310 uniquelyidentifies the requestor agent within a PCIe hierarchy. Accordingly,together with source ID 310, local transaction identifier 308 fieldprovides global identification of a transaction within a hierarchydomain.

Attributes field 304 specifies characteristics and relationships of thetransaction. In this regard, attributes field 304 is potentially used toprovide additional information that allows modification of the defaulthandling of transactions. In one embodiment, attributes field 304includes priority field 312, reserved field 314, ordering field 316, andno-snoop field 318. Here, priority sub-field 312 may be modified by aninitiator to assign a priority to the transaction. Reserved attributefield 314 is left reserved for future, or vendor-defined usage. Possibleusage models using priority or security attributes may be implementedusing the reserved attribute field.

In this example, ordering attribute field 316 is used to supply optionalinformation conveying the type of ordering that may modify defaultordering rules. According to one example implementation, an orderingattribute of “0” denotes default ordering rules are to apply, wherein anordering attribute of “1” denotes relaxed ordering, wherein writes canpass writes in the same direction, and read completions can pass writesin the same direction. Snoop attribute field 318 is utilized todetermine if transactions are snooped. As shown, channel ID Field 306identifies a channel that a transaction is associated with.

Link Layer

Link layer 210, also referred to as data link layer 210, acts as anintermediate stage between transaction layer 205 and the physical layer220. In one embodiment, a responsibility of the data link layer 210 isproviding a reliable mechanism for exchanging Transaction Layer Packets(TLPs) between two components a link. One side of the Data Link Layer210 accepts TLPs assembled by the Transaction Layer 205, applies packetsequence identifier 211, i.e. an identification number or packet number,calculates and applies an error detection code, i.e. CRC 212, andsubmits the modified TLPs to the Physical Layer 220 for transmissionacross a physical to an external device.

Physical Layer

In one embodiment, physical layer 220 includes logical sub block 221 andelectrical sub-block 222 to physically transmit a packet to an externaldevice. Here, logical sub-block 221 is responsible for the “digital”functions of Physical Layer 221. In this regard, the logical sub-blockincludes a transmit section to prepare outgoing information fortransmission by physical sub-block 222, and a receiver section toidentify and prepare received information before passing it to the LinkLayer 210.

The physical electrical sub-block 222 includes a transmitter and areceiver. The transmitter is supplied by logical sub-block 221 withsymbols, which the transmitter serializes and transmits onto to anexternal device. The receiver is supplied with serialized symbols froman external device and transforms the received signals into abit-stream. The bit-stream is de-serialized and supplied to logicalsub-block 221. In one embodiment, an 8b/10b transmission code isemployed, where ten-bit symbols are transmitted/received. Here, specialsymbols are used to frame a packet with frames 223. In addition, in oneexample, the receiver also provides a symbol clock recovered from theincoming serial stream.

As stated above, although transaction layer 205, link layer 210, andphysical layer 220 are discussed in reference to a specific embodimentof a PCIe protocol stack, a layered protocol stack is not so limited. Infact, any layered protocol may be included/implemented. As an example,an port/interface that is represented as a layered protocol includes:(1) a first layer to assemble packets, i.e. a transaction layer; asecond layer to sequence packets, i.e. a link layer; and a third layerto transmit the packets, i.e. a physical layer. As a specific example, acommon standard interface (CSI) layered protocol is utilized.

Referring next to FIG. 4, an embodiment of a PCIe serial point to pointfabric is illustrated. Although an embodiment of a PCIe serialpoint-to-point link is illustrated, a serial point-to-point link is notso limited, as it includes any transmission path for transmitting serialdata. In the embodiment shown, a basic PCIe link includes two,low-voltage, differentially driven signal pairs: a transmit pair 406/411and a receive pair 412/407. Accordingly, device 405 includestransmission logic 406 to transmit data to device 410 and receivinglogic 407 to receive data from device 410. In other words, twotransmitting paths, i.e. paths 416 and 417, and two receiving paths,i.e. paths 418 and 419, are included in a PCIe link.

A transmission path refers to any path for transmitting data, such as atransmission line, a copper line, an optical line, a wirelesscommunication channel, an infrared communication link, or othercommunication path. A connection between two devices, such as device 405and device 410, is referred to as a link, such as link 415. A link maysupport one lane —each lane representing a set of differential signalpairs (one pair for transmission, one pair for reception). To scalebandwidth, a link may aggregate multiple lanes denoted by xN, where N isany supported Link width, such as 1, 2, 4, 8, 12, 16, 20, 24, 32, 64, orwider.

A differential pair refers to two transmission paths, such as lines 416and 417, to transmit differential signals. As an example, when line 416toggles from a low voltage level to a high voltage level, i.e. a risingedge, line 417 drives from a high logic level to a low logic level, i.e.a falling edge. Differential signals potentially demonstrate betterelectrical characteristics, such as better signal integrity, i.e.cross-coupling, voltage overshoot/undershoot, ringing, etc. This allowsfor better timing window, which enables faster transmission frequencies.

In one implementation, as shown in FIG. 5, Physical layer 505 a,b, canbe responsible for the fast transfer of information on the physicalmedium (electrical or optical etc.). The physical link can bepoint-to-point between two Link layer entities, such as layer 505 a and505 b. The Link layer 510 a,b can abstract the Physical layer 505 a,bfrom the upper layers and provides the capability to reliably transferdata (as well as requests) and manage flow control between two directlyconnected entities. The Link Layer can also be responsible forvirtualizing the physical channel into multiple virtual channels andmessage classes. The Protocol layer 520 a,b relies on the Link layer 510a,b to map protocol messages into the appropriate message classes andvirtual channels before handing them to the Physical layer 505 a,b fortransfer across the physical links. Link layer 510 a,b may supportmultiple messages, such as a request, snoop, response, writeback,non-coherent data, among other examples.

A Physical layer 505 a,b (or PHY) can be implemented above theelectrical layer (i.e. electrical conductors connecting two components)and below the link layer 510 a,b, as illustrated in FIG. 5. The Physicallayer and corresponding logic can reside on each agent and connect thelink layers on two agents or nodes (A and B) separated from each other(e.g. on devices on either side of a link). The local and remoteelectrical layers are connected by physical media (e.g. wires,conductors, optical, etc.). The Physical layer 505 a,b, in oneembodiment, has two major phases, initialization and operation. Duringinitialization, the connection is opaque to the link layer and signalingmay involve a combination of timed states and handshake events. Duringoperation, the connection is transparent to the link layer and signalingis at a speed, with all lanes operating together as a single link.During the operation phase, the Physical layer transports flits 535 fromagent A to agent B and from agent B to agent A. The connection is alsoreferred to as a link and abstracts some physical aspects includingmedia, width and speed from the link layers while exchanging flits andcontrol/status of current configuration (e.g. width) with the linklayer. The initialization phase includes minor phases e.g. Polling,Configuration. The operation phase also includes minor phases (e.g. linkpower management states).

In one embodiment, Link layer 510 a,b can be implemented so as toprovide reliable data transfer between two protocol or routing entities.The Link layer can abstract Physical layer 505 a,b from the Protocollayer 520 a,b, and can be responsible for the flow control between twoprotocol agents (A, B), and provide virtual channel services to theProtocol layer (Message Classes) and Routing layer (Virtual Networks).The interface between the Protocol layer 520 a,b and the Link Layer 510a,b can typically be at the packet level. In one embodiment, thesmallest transfer unit at the Link Layer is referred to as a flit whicha specified number of bits, such as 192 bits or some other denomination.The Link Layer 510 a,b relies on the Physical layer 505 a,b to frame thePhysical layer's 505 a,b unit of transfer (phit 540) into the LinkLayer's 510 a,b unit of transfer (flit). In addition, the Link Layer 510a,b may be logically broken into two parts, a sender and a receiver. Asender/receiver pair on one entity may be connected to a receiver/senderpair on another entity. Flow Control is often performed on both a flitand a packet basis. Error detection and correction is also potentiallyperformed on a flit level basis.

In one embodiment, Routing layer 515 a,b can provide a flexible anddistributed method to route transactions from a source to a destination.The scheme is flexible since routing algorithms for multiple topologiesmay be specified through programmable routing tables at each router (theprogramming in one embodiment is performed by firmware, software, or acombination thereof). The routing functionality may be distributed; therouting may be done through a series of routing steps, with each routingstep being defined through a lookup of a table at either the source,intermediate, or destination routers. The lookup at a source may be usedto inject a packet into the fabric. The lookup at an intermediate routermay be used to route an packet from an input port to an output port. Thelookup at a destination port may be used to target the destinationprotocol agent. Note that the Routing layer, in some implementations,can be thin since the routing tables, and, hence the routing algorithms,are not specifically defined by specification. This allows forflexibility and a variety of usage models, including flexible platformarchitectural topologies to be defined by the system implementation. TheRouting layer 515 a,b relies on the Link layer 510 a,b for providing theuse of up to three (or more) virtual networks (VNs)—in one example, twodeadlock-free VNs, VN0 and VN1 with several message classes defined ineach virtual network. A shared adaptive virtual network (VNA) may bedefined in the Link layer, but this adaptive network may not be exposeddirectly in routing concepts, since each message class and virtualnetwork may have dedicated resources and guaranteed forward progress,among other features and examples.

In one embodiment, Protocol Layer 520 a,b can provide a CoherenceProtocol to support agents caching lines of data from memory. An agentwishing to cache memory data may use the coherence protocol to read theline of data to load into its cache. An agent wishing to modify a lineof data in its cache may use the coherence protocol to acquire ownershipof the line before modifying the data. After modifying a line, an agentmay follow protocol requirements of keeping it in its cache until iteither writes the line back to memory or includes the line in a responseto an external request. Lastly, an agent may fulfill external requeststo invalidate a line in its cache. The protocol ensures coherency of thedata by dictating the rules all caching agents may follow. It alsoprovides the means for agents without caches to coherently read andwrite memory data.

Physical layers of existing interconnect and communicationarchitectures, including PCIe, can be leveraged to provide shared memoryand I/O services within a system. Traditionally, cacheable memory cannotbe shared between independent systems using traditional load/store(LD/ST) memory semantics. An independent system, or “node”, can beindependent in the sense that it functions as a single logical entity,is controlled by a single operating system (and/or single BIOS orVirtual Machine Monitor (VMM)), and/or has an independent fault domain.A single node can include one or multiple processor devices, beimplemented on a single board or multiple boards, and include localmemory, including cacheable memory that can be accessed using LD/STsemantics by the devices on the same node. Within a node, shared memorycan include one or more blocks of memory, such as a random access memory(RAM), that can be accessed by several different processors (e.g.,central processing units (CPUs)) within a node. Shared memory can alsoinclude the local memory of the processors or other devices in the node.The multiple devices within a node having shared memory can share asingle view of data within the shared memory. I/O communicationinvolving shared memory can be very low latency and allow quick accessto the memory by the multiple processors.

Traditionally, memory sharing between different nodes has not allowedmemory sharing according to a load/store paradigm. For instance, in somesystems, memory sharing between different nodes has been facilitatedthrough distributed memory architectures. In traditional solutions,computational tasks operate on local data, and if data of another nodeis desired, the computational task (e.g., executed by another CPU node)communicates with the other node, for instance, over a communicationchannel utilizing a communication protocol stack, such as Ethernet,InfiniBand, or another layered protocol. In traditional multi-nodesystems, the processors of different nodes do not have to be aware wheredata resides. Sharing data using traditional approaches, such as over aprotocol stack, can have a significantly higher latency than memorysharing within a node using a load/store paradigm. Rather than directlyaddressing and operating on data in shared memory, one node can requestdata from another using an existing protocol handshake such as Ethernet(or Infiniband), and the source node can provide the data, such that thedata can be stored and operated on by the requesting node, among otherexamples.

In some implementations, a shared memory architecture can be providedthat allows memory to be shared between independent nodes for exclusiveor shared access using load/store (LD/ST) memory semantics. In oneexample, memory semantics (and directory information, if applicable)along with I/O semantics (for protocols such as PCIe) can be exported oneither a common set of pins or a separate set of pins. In such a system,the improved shared memory architecture can each of a plurality of nodesin a system to maintain its own independent fault domain (and localmemory), while enabling a shared memory pool for access by the nodes andlow-latency message passing between nodes using memory according toLD/ST semantics. In some implementations, such a shared memory pool canbe dynamically (or statically) allocated between different nodes.Accordingly, one can also configure the various nodes of a system intodynamically changing groups of nodes to work cooperatively and flexiblyon various tasks making use of the shared memory infrastructure, forinstance, as demand arises.

In some implementations, the shared memory architecture can be based ona buffered memory interface. The buffered memory interface, itself, canbe based on a general purpose input/output (GPIO) interconnect interfaceand protocol. For instance, the physical and link layer definitions ofthe GPIO interconnect can also be implemented in the buffered memoryprotocol. Indeed, logic used to support the physical and link layers ofthe GPIO protocol can be reused at interfaces supporting the bufferedmemory protocol. The buffered memory protocol can also share messageclasses, such as a request, response, and writeback message class, amongother examples. While opcode values within the buffered memory protocolmessage can be interpreted differently than in the GPIO protocol, thesame general packet and flit formats can be utilized in both thebuffered memory protocol and the GPIO interconnect upon which it isbuilt.

In one example, a flit format can be defined for flits to be sentbetween agents in the GPIO protocol. FIG. 6 illustrates a representation600 of a generalized flit for an 8 lane link width. Each column of therepresentation 600 can symbolize a link lane and each row a respectiveUI. In some implementations, a single flit can be subdivided into two ormore slots. Distinct messages or link layer headers can be included ineach slot, allowing multiple distinct, and in some cases, independentmessages corresponding to potentially different transactions to be sentin a single flit. Further, the multiple messages included in slots of asingle flit may also be destined to different destination nodes, amongother examples. For instance, the example of FIG. 6 illustrates a flitformat with three slots. The shaded portions can represent the portionof the flit included in a respective slot.

In the example of FIG. 6, three slots, Slots 0, 1, and 2, are provided.Slot 0 can be provided 72 bits of flit space, of which 22 bits arededicated to message header fields and 50 bits to message payload space.Slot 1 can be provided with 70 bits of flit space, of which 20 bits arededicated to message header fields and 50 bits to message payload space.The difference in message header field space between can be optimized toprovide that certain message types will be designated for inclusion inSlot 0 (e.g., where more message header encoding is utilized). A thirdslot, Slot 2, can be provided that occupies substantially less spacethan Slots 0 and 1, in this case utilizing 18 bits of flit space. Slot 2can be optimized to handle those messages, such as acknowledges, creditreturns, and the like that do no utilize larger message payloads.Additionally, a floating payload field can be provided that allows anadditional 11 bits to be alternatively applied to supplement the payloadfield of either Slot 0 or Slot 1.

Continuing with the specific example of FIG. 6, other fields can beglobal to a flit (i.e., apply across the flit and not to a particularslot). For instance, a header bit can be provided together with a 4-bitflit control field that can be used to designate such information as avirtual network of the flit, identify how the flit is to be encoded,among other examples. Additionally, error control functionality can beprovided, such as through a 16-bit cyclic CRC field, among otherpotential examples.

A flit format can be defined so as to optimize throughput of messages onthe Link layer. Some traditional protocols have utilized unslotted,smaller flits. For instance, in QPI an 80-bit flit was utilized. Whilethe flit throughput of a larger (e.g., 192-bit flit) may be lower,message or packet throughput can be increased by optimizing use of theflit data. For instance, in some protocols, an entire flit space (e.g.,80-bit) was utilized regardless of the message size or type. Bysubdividing a larger flit into slots of predetermined lengths andfields, the 192 flit length can be optimized realizing higher efficiencyeven in instances when one or more of the available slots are sometimesunused. Indeed, Link layer traffic can be assumed to include manydifferent types of messages and traffic, including messages and packetswith varying header lengths and fields. The respective lengths andorganization of slots defined in a flit can be defined so as tocorrespond with the statistical or expected frequency of variousmessages and the needs of these messages. For instance, two larger slotscan be defined for every small slot, to accommodate an expectedstatistical frequency of messaging using these larger message types andheader lengths, among other example. Further, flexibility can also beprovided to further accommodate the varied traffic, such as through afloating payload field, as in the example of FIG. 6. In some instances,a flit format can be fixed, including the bits dedicated to particularslots in the flit.

In the example of FIG. 6, a “Hdr” field can be provided for the flitgenerally and represent a header indication for the flit. In someinstances, the Hdr field can indicate whether the flit is a header flitor a data flit. In data flits, the flit can still remain slotted, butomit or replace the use of certain fields with payload data. In somecases, data fields may include an opcode and payload data. In the caseof header flits, a variety of header fields can be provided. In theexample of FIG. 6, “Oc” fields can be provided for each slot, the Ocfield representing an opcode. Similarly, one or more slots can have acorresponding “msg” field representing a message type of thecorresponding packet to be included in the slot, provided the slot isdesigned to handle such packet types, etc. “DNID” fields can represent aDestination Node ID, a “TID” field can represent a transaction ortracker ID, a “RHTID” field can represent either a requestor node ID ora home tracker ID, among other potential fields. Further, one or moreslots can be provided with payload fields. Additionally, a CRC field canbe included within a flit to provide a CRC value for the flit, amongother examples.

The multi-slotted flit of a GPIO protocol can be reused by a bufferedmemory protocol. FIG. 7 shows a simplified block diagram 700illustrating an example topology of a computing system including CPUdevices 705, 710 interconnected by an GPIO interconnect link. Each CPU705, 710 can be likewise connected to one or more respective bufferdevices 715 a-1 using corresponding buffered memory protocol links(“MemLink”). Each buffer device can implement a memory controller forsystem memory of the system. As noted above, in some implementations,the buffered memory protocol interconnect can be based on the GPIOprotocol, in that the physical and link layers of the buffered memoryprotocols are based on the same physical and link layer definitions ofthe GPIO protocol. Although not illustrated in FIG. 7, the CPUs 705, 710can be further connected to one or more downstream devices using theGPIO protocol.

As further shown in the example of FIG. 7, buffer devices 715 a-1 can beconnected to memory devices, such as dual in-line memory module (DIMM)devices. The memory corresponding to each buffer device can beconsidered local to the CPU (e.g., 705, 701) to which the buffer deviceis connected. However, other devices (including the other CPU) canaccess the memory by other sockets using GPIO protocol-compliant links.In some implementations, a port running the buffered memory protocol mayonly support the commands for communicating with the memory and onlysupport the buffered memory protocol (i.e., not the GPIO protocol andthe buffered memory protocol). Additionally, in some implementations,the GPIO interconnect protocol may support routing and indicate suchinformation (e.g., in its packets) such as the requesting anddestination node identifiers. The buffered memory protocol, on the otherhand, may be a point-to-point interface that does not utilize routing.Consequently, some fields used in the GPIO protocol may be dispensedwith in packets sent using the buffered memory interfaces. Instead,fields can be designated for use in carrying address decode informationhost to buffer, among other examples.

In further implementations, buffer devices 715 a-1 can support a twolevel memory topology with some amount of fast memory (e.g., DRAM)serving as a cache for a larger, slower memory (e.g., non-volatilememory). In one such implementation, one or more of the buffer devices715 a-1 can use DDR as near, fast memory and transactional DDR DIMMs asthe larger “far” memory, among other examples. Transactional DIMMs canutilize protocols (e.g., DDR-Transactional (DDR-T)) to communicate to avolatile memory single in-line memory module (SIMM) using transactionalcommands.

The buffered memory protocol and systems utilizing a buffered memoryprotocol (such as those illustrated above) can be extended to enable ashared memory architecture that allows memory to be shared betweenindependent nodes for exclusive or shared access using load/store(LD/ST) memory semantics. Turning to FIG. 8A, a simplified block diagram800 a is shown illustrating an example system including shared memory805 capable of being accessed using load/store techniques by each of aplurality of independent nodes 810 a-810 n. For instance, a sharedmemory controller 815 can be provided that can accept load/store accessrequests of the various nodes 810 a-810 n on the system. Shared memory805 can be implemented utilizing synchronous dynamic random accessmemory (SDRAM), dual in-line memory modules (DIMM), and othernon-volatile memory (or volatile memory).

Each node may itself have one or multiple CPU sockets and may alsoinclude local memory that remains insulated from LD/ST access by othernodes in the system. The node can communicate with other devices on thesystem (e.g., shared memory controller 815, networking controller 820,other nodes, etc.) using one or more protocols, including PCIe, QPI,Ethernet, among other examples. In some implementations, a shared memorylink (SML) protocol can be provided through which low latency LD/STmemory semantics can be supported. SML can be used, for instance, incommunicating reads and writes of shared memory 805 (through sharedmemory controller 815) by the various nodes 810 a-810 n of a system.

In one example, SML can be based on a memory access protocol, such asScalable Memory Interconnect (SMI) 3rd generation (SMI3). Other memoryaccess protocols can be alternatively used, such as transactional memoryaccess protocols such as fully buffered DIMM (FB-DIMM), DDRTransactional (DDR-T), among other examples. In other instances, SML canbe based on native PCIe memory read/write semantics with additionaldirectory extensions. A memory-protocol-based implementation of SML canoffer bandwidth efficiency advantages due to being tailored to cacheline memory accesses. While high performance inter-device communicationprotocols exist, such as PCIe, upper layers (e.g., transaction and linklayers) of such protocols can introduce latency that degradesapplication of the full protocol for use in LD/ST memory transactions,including transactions involving a shared memory 805. A memory protocol,such as SMI3, can allow a potential additional advantage of offeringlower latency accesses since it can bypass most of another protocolstack, such as PCIe. Accordingly, implementations of SML can utilizeSMI3 or another memory protocol running on a logical and physical PHY ofanother protocol, such as SMI3 on PCIe.

As noted, in some implementation, a shared memory controller (SMC) 815can be provided that includes logic for handling load/store requests ofnodes 810 a-810 n in the system. Load/store requests can be received bythe SMC 815 over links utilizing SML and connecting the nodes 810 a-810n to the SMC 815. In some implementations the SMC 815 can be implementedas a device, such as an application-specific integrated circuit (ASIC),including logic for servicing the access requests of the nodes 810 a-810n for shared memory resources. In other instances, the SMC 815 (as wellas shared memory 805) can reside on a device, chip, or board separatefrom one or more (or even all) of the nodes 810 a-810 n. The SMC 815 canfurther include logic to coordinate various nodes' transactions thatinvolve shared memory 805. Additionally, the SMC can maintain adirectory tracking access to various data resources, such as each cacheline, included in shared memory 805. For instance, a data resource canbe in a shared access state (e.g., capable of being accessed (e.g.,loaded or read) by multiple processing and/or I/O devices within a node,simultaneously), an exclusive access state (e.g., reserved exclusively,if not temporarily, by a single processing and/or I/O device within anode (e.g., for a store or write operation), an uncached state, amongother potential examples. Further, while each node may have directaccess to one or more portions of shared memory 805, differentaddressing schemes and values may be employed by the various nodes(e.g., 810 a-810 n) resulting in the same shared memory data beingreferred to (e.g., in an instruction) by a first node according to afirst address value and a second node being referring to the same databy a second address value. The SMC 815 can include logic, including datastructures mapping nodes' addresses to shared memory resources, to allowthe SMC 815 to interpret the various access requests of the variousnodes.

Additionally, in some cases, some portion of shared memory (e.g.,certain partitions, memory blocks, records, files, etc.) may be subjectto certain permissions, rules, and assignments such that only a portionof the nodes 810 a-810 n are allowed (e.g., by the SMC 815) to accessthe corresponding data. Indeed, each shared memory resource may beassigned to a respective (and in some cases different) subset of thenodes 810 a-810 n of the system. These assignments can be dynamic andSMC 815 can modify such rules and permissions (e.g., on-demand,dynamically, etc.) to accommodate new or changed rules, permissions,node assignments and ownership applicable to a given portion of theshared memory 805.

An example SMC 815 can further track various transactions involvingnodes (e.g., 810 a-810 n) in the system accessing one or more sharedmemory resources. For instance, SMC 815 can track information for eachshared memory 805 transaction, including identification of the node(s)involved in the transaction, progress of the transaction (e.g., whetherit has been completed), among other transaction information. This canpermit some of the transaction-oriented aspects of traditionaldistributed memory architectures to be applied to the improvedmulti-node shared memory architecture described herein. Additionally,transaction tracking (e.g., by the SMC) can be used to assist inmaintaining or enforcing the distinct and independent fault domains ofeach respective node. For instance, the SMC can maintain thecorresponding Node ID for each transaction-in-progress in its internaldata structures, including in memory, and use that information toenforce access rights and maintain individual fault-domains for eachnode. Accordingly, when one of the nodes goes down (e.g., due to acritical error, triggered recovery sequence, or other fault or event),only that node and its transactions involving the shared memory 805 areinterrupted (e.g., dumped by the SMC)—transactions of the remainingnodes that involve the shared memory 805 continue on independent of thefault in the other node.

A system can include multiple nodes. Additionally, some example systemscan include multiple SMCs. In some cases, a node may be able to accessshared memory off a remote SMC to which it is not directly attached to(i.e., the node's local SMC connects to the remote SMC through one ormultiple SML Link hops). The remote SMC may be in the same board orcould be in a different board. In some cases, some of the nodes may beoff-system (e.g., off board or off chip) but nonetheless access sharedmemory 805. For instance, one or more off-system nodes can connectdirectly to the SMC using an SML-compliant link, among other examples.Additionally, other systems that include their own SMC and shared memorycan also connect with the SMC 810 to extend sharing of memory 805 tonodes included, for instance, on another board that interface with theother SMC connected to the SMC over an SML link. Still further, networkconnections can be tunneled through to further extend access to otheroff-board or off-chip nodes. For instance, SML can tunnel over anEthernet connection (e.g., provided through network controller 820)communicatively coupling the example system of FIG. 8A with anothersystem that can also include one or more other nodes and allow thesenodes to also gain access to SMC 815 and thereby shared memory 805,among other examples.

As another example, as shown in the simplified block diagram 800 b ofFIG. 8B, an improved shared memory architecture permitting shared accessby multiple independent nodes according to a LD/ST memory semantic canflexibly allow for the provision of a variety of different multi-nodesystem designs. Various combinations of the multiple nodes can beassigned to share portions of one or more shared memory blocks providedin an example system. For instance, another example system shown in theexample of FIG. 8B, can include multiple devices 850 a-850 dimplemented, for instance, as separate dies, boards, chips, etc., eachdevice including one or more independent CPU nodes (e.g., 810 a-810 h).Each node can include its own local memory. One or more of the multipledevices 850 a-850 d can further include shared memory that can beaccessed by two or more of the nodes 810 a-810 h of the system.

The system illustrated in FIG. 8B is an example provided to illustratesome of the variability that can be realized through an improved sharedmemory architecture, such as shown and described herein. For instance,each of a Device A 850 a and Device C 850 c can include a respectiveshared memory element (e.g., 805 a, 805 b). Accordingly, in someimplementations, each shared memory element on a distinct device mayfurther include a respective shared memory controller (SMC) 815 a, 815b. Various combinations of nodes 810 a-810 h can be communicativelycoupled to each SMC (e.g., 815 a, 815 b) allowing the nodes to accessthe corresponding shared memory (e.g., 805 a, 805 b). As an example, SMC815 a of Device A 850 a can connect to nodes 810 a, 810 b on Device Ausing a direct data link supporting SML. Additionally, another node 810c on another device (e.g., Device C 850 c) can also have access to theshared memory 805 a by virtue of a direct, hardwired connection(supporting SML) from the node 810 c (and/or its device 850 c) to SMC815 a. Indirect, network-based, or other such connections can also beused to allow nodes (e.g., 810 f-810 h) of a remote or off-board device(e.g., Device D 850 d) to utilize a conventional protocol stack tointerface with SMC 815 a to also have access to shared memory 805 a. Forinstance, an SML tunnel 855 can be established over an Ethernet, InfiniBand, or other connection coupling Device A and Device D. Whileestablishing and maintaining the tunnel can introduce some additionaloverhead and latency, compared to SML running on otherless-software-managed physical connections, the SML tunnel 855 whenestablished can operate as other SML channels and allow the nodes 810f-810 h to interface with SMC 815 a over SML and access shared memory805 a as any other node communicating with SMC over an SML link can. Forinstance, reliability and ordering of the packets in the SML channelscan be enforced either by the networking components in the system or itcan be enforced end-to-end between the SMCs.

In still other examples, nodes (e.g., 815 d, 815 e) on a devicedifferent from that hosting a particular portion of shared memory (e.g.,805 a) can connect indirectly to the corresponding SMC (e.g., SMC 815 a)by connecting directly to another SMC (e.g., 815 b) that is itselfcoupled (e.g., using an SML link) to the corresponding SMC (e.g., 815a). Linking two or more SMCs (e.g., 815 a, 815 b) can effectively expandthe amount of shared memory available to the nodes 810 a-810 h on thesystem. For instance, by virtue of a link between SMCs 815 a, 815 b inthe example of FIG. 8B, in some implementations, any of the nodes (e.g.,810 a-810 c, 810 f-810 h) capable of accessing shared memory 805 athrough SMC 815 a may also potentially access sharable memory 805 b byvirtue of the connection between SMC 815 a and SMC 815 b. Likewise, insome implementations, each of the nodes directly accessing SMC 815 b canalso access sharable memory 805 a by virtue of the connection betweenthe SMCs 815 a, 815 b, among other potential examples.

As noted, independent nodes can each access shared memory, includingshared memory included in memory not connected to the SMC to which thenode is directly connected. The shared memory is effectively pooled.While a traditional buffered memory protocol can assume point-to-pointcommunication, the pooling of shared memory and joint management of thismemory by multiple SMCs can involve packets and flits relating to thismemory to traverse multiple hops and SMCs before they arrive to theirintended destination. In this respect, the multiple SMCs can form anetwork of SMCs and each SMC can include logic for determining how toroute a particular flit from its directly connected nodes to the SMCconnected to the memory addressed by the flit. For instance, in FIG. 8C,an example 800 c is shown of multiple SMCs 815 a-c interconnected witheach other SMC by one or more SML links. Each SMC can be connected to asubset of the processor nodes in the system. Further, each SMC candirectly connect to and provide access to a respective subset of thememory elements that compose the shared memory pool. As an example, SMC815 a can connect to nodes 810 a, 810 b and shared memory elements(e.g., 805 a). Another node 810 i can access a line of memory stored inshared memory portion 805 a by sending a request to SMC 815 c which canroute the request, over an SML link to SMC 815 a. SMC 815 a can manage amemory action in connection with the request and respond, in some cases,by providing read data, an acknowledgement, or other information to thenode 810 i by routing the response over an SML link to SMC 815 c.Instead of routing SML communications directly between SMC 815 a and 815c, in other instances, the communications can additionally be routedover other SMCs (e.g., 815 b). Accordingly, each SMC in a shared memoryarchitecture can include routing logic, implemented in hardware and/orsoftware to facilitate routing communications between SMCs within thenetwork.

As noted above, an improved shared memory architecture can include alow-latency link protocol (i.e., SML) based on a memory access protocol,such as SMI3, and provided to facilitate load/store requests involvingthe shared memory. Whereas traditional SMI3 and other memory accessprotocols may be configured for use in memory sharing within a singlenode, SML can extend memory access semantics to multiple nodes to allowmemory sharing between the multiple nodes. Further, SML can potentiallybe utilized on any physical communication link. SML can utilize a memoryaccess protocol supporting LD/ST memory semantics that is overlaid on aphysical layer (and corresponding physical layer logic) adapted tointerconnect distinct devices (and nodes). Additionally, physical layerlogic of SML can provide for no packet dropping and error retryfunctionality, among other features.

In some implementations, SML can be can be implemented by overlayingSMI3 on a PCIe PHY. An SML link layer can be provided (e.g., in lieu ofa traditional PCIe link layer) to forego flow control and other featuresand facilitate lower latency memory access such as would becharacteristic in traditional CPU memory access architectures. In oneexample, SML link layer logic can multiplex between shared memorytransactions and other transactions. For instance, SML link layer logiccan multiplex between SMI3 and PCIe transactions. For instance, SMI3 (oranother memory protocol) can overlay on top of PCIe (or anotherinterconnect protocol) so that the link can dynamically switch betweenSMI3 and PCIe transactions. This can allow traditional PCIe traffic toeffectively coexist on the same link as SML traffic in some instances.

Turning to FIG. 9, a representation 900 is shown illustrating a firstimplementation of SML. For instance, SML can be implemented byoverlaying SMI3 on a PCIe PHY. The physical layer can use standard PCIe128 b/130 b encoding for all physical layer activities including linktraining as well as PCIe data blocks. SML can provide for traffic on thelanes (e.g., Lane0-Lane7) of the link to be multiplexed between PCIepackets and SMI3 flits. For example, in the implementation illustratedin FIG. 9, the sync header of the PCIe 128 b/130 b encoding can bemodified and used to indicate that SMI3 flits are to be sent on thelanes of the link rather than PCIe packets. In traditional PCIe 128b/130 b encoding, valid sync headers (e.g., 910) can include the sendingof either a 10b pattern on all lanes of the link (to indicate that thetype of payload of the block is to be PCIe Data Block) or a 01b patternon all lanes of the link (to indicate that the type of payload of theblock is to be PCIe Ordered Set Block). In an example of SML, analternate sync header can be defined to differentiate SMI3 flit trafficfrom PCIe data blocks and ordered sets. In one example, illustrated inFIG. 9, the PCIe 128 b/130 b sync header (e.g., 905 a, 905 b) can beencoded with alternating 01b, 10b patterns on odd/even lanes to identifythat SMI3 flits are to be sent. In another alternative implementation,the 128 b/130 b sync header encoding for SMI3 traffic can be defined byalternating 10b, 01b patterns on odd/even lanes, among other exampleencodings. In some cases, SMI3 flits can be transmitted immediatelyfollowing the SMI3 sync header on a per-byte basis, with the transitionbetween PCIe and SMI3 protocols taking place at the block boundary.

In some implementations, such as that illustrated in the example of FIG.9, the transition between the protocols can be defined to take place atthe block boundary irrespective of whether it corresponds to an SMI3flit or PCIe packet boundary. For instance, a block can be defined toinclude a predefined amount of data (e.g., 16 symbols, 128 bytes, etc.).In such implementations, when the block boundary does not correspond toan SMI3 flit or PCIe packet boundary, the transmission of an entire SMI3flit may be interrupted. An interrupted SMI3 flit can be resumed in thenext SMI3 block indicated by the sending of another sync header encodedfor SMI3.

Turning to FIG. 10A, a representation 1000 is shown illustrating anotherexample implementation of SML. In the example of FIG. 10A, rather thanusing a specialized sync header encoding to signal transitions betweenmemory access and interconnect protocol traffic, physical layer framingtokens can be used. A framing token (or “token”) can be a physical layerdata encapsulation that specifies or implies the number of symbols to beincluded in a stream of data associated with the token. Consequently,the framing token can identify that a stream is beginning as well asimply where it will end and can therefore be used to also identify thelocation of the next framing token. A framing token of a data stream canbe located in the first symbol (Symbol 0) of the first lane (e.g., Lane0) of the first data block of the data stream. In the example of PCIs,five framing tokens can be defined, including start of TLP traffic (STP)token, end of data stream (EDS) token, end bad (EDB) token, start ofDLLP (SDP) token, and logical idle (IDL) token.

In the example of FIG. 10A, SML can be implemented by overlaying (or“tunneling”) SMI3 or another data access protocol on PCIe and thestandard PCIe STP token can be modified to define a new STP token thatidentifies that SMI3 (instead of TLP traffic) is to commence on thelanes of the link. In some examples, values of reserve bits of thestandard PCIe STP token can be modified to define the SMI3 STP token inSML. Further, as shown in FIG. 10B, an STP token 1005 can includeseveral fields, including a 1010 field that identifies the length of theSMI3 payload (in terms of the number of flits) that is to follow. Insome implementations, one or more standard payload lengths can bedefined for TLP data. SMI3 data can, in some implementations, be definedto include a fixed number of flits, or in other cases, may have variablenumbers of flits in which case the length field for the number of SMI3flits becomes a field that can be disregarded. Further, the length fieldfor an SMI3 STP can be defined as a length other than one of the definedTLP payload lengths. Accordingly, an SMI3 STP can be identified based ona non-TLP length value being present in the STP length field, as oneexample. For example, in one implementation, the upper 3-bits of the11-bit STP length field can be set to 111b to indicate the SMI3 packet(e.g., based on the assumption that no specification-compliant PCIe TLPcan be long enough to have a length where the upper 3 bits of the lengthfield would result in l's). Other implementations can alter or encodeother fields of the STP token to differentiate a PCIe STP tokenidentifying a traditional PCIe TLP data payload from a SMI3 STP tokenidentifying that SMI3 data is encapsulated in TLP data.

Returning to the example of FIG. 10A, sync header data can follow theencoding specified for traditional PCIe 128 b/130 b encoding. Forinstance, at 1015 a-c, sync headers with value 10b are receivedindicating that data blocks are forthcoming. When a PCIe STP (e.g.,1020) is received, a PCIe TLP payload is expected and the data stream isprocessed accordingly. Consistent with the payload length identified inthe PCIe STP 1020, the PCIe TLP payload can utilize the full payloadlength allocated. Another STP token can be received essentially at anytime within a data block following the end of the TLP payload. Forinstance, at 1025, an SMI3 STP can be received signaling a transitionfrom PCIe TLP data to SMI3 flit data. The SMI3 STP can be sent, forinstance, as soon as an end of the PCIe packet data is identified.

Continuing with the example of FIG. 10A, as with PCIe TLP data, the SMI3STP 1025 can define a length of the SMI3 flit payload that is to follow.For instance, the payload length of the SMI3 data can correspond to thenumber of SMI3 flits in terms of DWs to follow. A window (e.g., endingat Symbol 15 of Lane 3) corresponding to the payload length can therebybe defined on the lanes, in which only SMI3 data is to be sent duringthe window. When the window concludes, other data can be sent, such asanother PCIe STP to recommence sending of TLP data or other data, suchas ordered set data. For instance, as shown in the example of FIG. 10A,an EDS token is sent following the end of the SMI3 data window definedby SMI3 STP token 1025. The EDS token can signal the end of the datastream and imply that an ordered set block is to follow, as is the casein the example of FIG. 10A. A sync header 1040 is sent that is encoded01b to indicate that an ordered set block is to be sent. In this case aPCIe SKP ordered set is sent. Such ordered sets can be sent periodicallyor according to set intervals or windows such that various PHY-leveltasks and coordination can be performed, including initializing bitalignment, initializing symbol alignment, exchanging PHY parameters,compensating for different bit rates for two communicating ports, amongother examples. In some cases, a mandated ordered set can be sent tointerrupt a defined window or data block specified for SMI3 flit data bya corresponding SMI3 STP token.

While not shown explicitly in the example of FIG. 10A, an STP token canalso be used to transition from SMI3 flit data on the link to PCIe TLPdata. For instance, following the end of a defined SMI3 window, a PCIeSTP token (e.g., similar to token 1020) can be sent to indicate that thenext window is for the sending of a specified amount of PCIe TLP data.

Memory access flits (e.g., SMI3 flits) may vary in size in someembodiments, making it difficult to predict, a priori, how much data toreserve in the corresponding STP token (e.g., SMI3 STP token) for thememory access payload. As an example, as shown in FIG. 10, SMI3 STP 1025can have a length field indicating that 244 bytes of SMI3 data is to beexpected following the SMI3 STP 1025. However, in this example, only tenflits (e.g., SMI3 Flits 0-9) are ready to be sent during the window andthese ten SMI3 flits only utilize 240 of the 244 bytes. Accordingly,four (4) bytes of empty bandwidth is left, and these are filled with IDLtokens. This can be particularly suboptimal when PCIe TLP data is queuedand waiting for the SMI3 window to close. In other cases, the windowprovided for the sending of SMI3 flits may be insufficient to send theamount of SMI3 data ready for the lane. Arbitration techniques can beemployed to determine how to arbitrate between SMI3 and PCIe TLP datacoexisting on the link. Further, in some implementations, the length ofthe SMI3 windows can be dynamically modified to assist in more efficientuse of the link. For instance, arbitration or other logic can monitorhow well the defined SMI3 windows are utilized to determine whether thedefined window length can be better optimized to the amount of SMI3 (andcompeting PCIe TLP traffic) expected for the lane. Accordingly, in suchimplementations, the length field values of SMI3 STP tokens can bedynamically adjusted (e.g., between different values) depending on theamount of link bandwidth that SMI3 flit data should be allocated (e.g.,relative to other PCIe data, including TLP, DLLP, and ordered set data),among other examples.

Turning to FIG. 11, a representation 1100 of another exampleimplementation of SML is illustrated. In this alternative embodiment,SML can provide for interleaving SMI3 and PCIe protocols through amodified PCIe framing token. As noted above, an EDS token can be used inPCIe to indicate an end of a data stream and indicate that the nextblock will be an ordered set block. In the example of FIG. 11, SML candefine an SMI3 EDS token (e.g., 1105) that indicates the end of a TLPdata stream and the transition to SMI3 flit transmissions. An SMI3 EDS(e.g., 1105) can be defined by encoding a portion of the reserved bitsof the traditional EDS token to indicate that SMI3 data is to follow,rather than PCIe ordered sets or other data that is to follow a PCIeEDS. Unlike the traditional EDS token, the SMI3 EDS can be sent atessentially anywhere within a PCIe data block. This can permitadditional flexibility in sending SMI3 data and accommodatingcorresponding low-latency shared memory transactions. For instance, atransition from PCIe to SMI3 can be accomplished with a single doubleword (DW) of overhead. Further, as with traditional EDS tokens, anexample SMI3 EDS may not specify a length associated with the SMI3 datathat is to follow the token. Following an SMI3 EDS, PCIe TLP data canconclude and SMI3 flits proceed on the link. SMI3 traffic can proceeduntil SMI3 logic passes control back to PCIe logic. In someimplementations, the sending of an SMI3 EDS causes control to be passedfrom PCIe logic to SMI3 logic provided, for instance, on devicesconnected on the link.

In one example, SMI3 (or another protocol) can define its own linkcontrol signaling for use in performing link layer control. For example,in one implementation, SML can define a specialized version of a SMI3link layer control (LLCTRL) flit (e.g., 1110) that indicates atransition from SMI3 back to PCIe protocol. As with an SMI3 EDS, thedefined LLCTRL flit (e.g., 1110) can cause control to be passed fromSMI3 logic back to PCIe logic. In some cases, as shown in the example ofFIG. 11, the defined LLCTRL flit (e.g., 1110) can be padded with apredefined number of LLCTRL idle (LLCTRL-IDLE) flits (e.g., 1115) beforecompleting the transition to PCIe. For instance, the number ofLLCTRL-IDLE flits 1115 to be sent to pad the SMI3 LLCTRL flit 1110 candepend on the latency to decode the defined SMI3 LLCTRL flit 1110signaling the transition. After completing the transition back to PCIe,an STP packet can be sent and TLP packet data can recommence on the linkunder control of PCIe.

As noted in connection with the examples of FIGS. 8A-8C, shared memorycontrollers in a shared buffered memory architecture can include routinglogic, implemented in hardware and/or software, to route communicationsover SML links between SMCs to facilitate data transactions involvingnodes and shared memory. To facilitate routing of flits among thenetwork of SMCs, additional fields can be provided in SML flits toassist in routing requests and responses to those requests to theirappropriate destinations. Table 1 describes example fields that can beprovided in a flit to accommodate routing in a multi-SMC shared memoryarchitecture.

TABLE 1 Alternate Fields for SML Flits PLM Packet Field DeltasDescription Source SMC Encodes the source SMC and internal SMC LinkID[8:0] agent for a request Address[59:52] Upper bits of the globalsystem address Slot-2 Mode Bit When set, indicates that a particularslot in a defined multi-slot flit format has been repurposed to carrysome of the upper address bits. When clear, the particular slot carriesinformation in accordance with a buffered memory link protocol.Alternate Route Specifies that a request should take an alternateRequested (ARR) route through fabric to its destination

DIn some implementations, as introduced in Table 1, a flit can beprovided with a field to indicate the source (and, in some cases, alsothe destination) node identifier of a given transaction. A link ID fieldcan be encoded with bits to identify an identity of the corresponding aswell as some additional bits to identify the particular link (or SMCagent) over which the request is sent/received SMC. The SMC identifiercan be a unique value within the system. In instances where the link IDvalue is a concatenation, combination, or otherwise based on both theSMC's ID and a particular link's ID, it can be expected that the valueof the link ID field is globally unique within the system. A similarfield can be provided, in some implementations, to indicate thedestination SMC for the request (although, in other implementations, thedestination SMC may be identifiable from the memory address that is thesubject of the transaction). Further, internally generated transactionsby a particular SMC for events such as memory migration, reliability-,accessibility-, serviceability-(RAS) related memory read/write/flush,etc., can have one or more unique “Link numbers” even though they do notreside off an external Link. The Link ID field can be processed by areceiving SMC and interpreted to determine how to route the flit to/fromits destination either locally (between nodes and SMCs that are directlyconnected) or over one or more additional SMCs and one or more SML“hops”.

By pooling memory of multiple memory elements managed by multipledifferent SMCs, the size of the address space can be significantlylarger than would be anticipated for point-to-point transactions betweena node and a single, corresponding memory controller. For a pooledmemory architecture, a larger address space can be made available tosupport the needs of all the nodes (e.g., in a rack or server tower),which will be the aggregated amount of memory for each node in the pool.In some cases, each node's private memory may likely reside off the SMCdirectly attached, this is not a requirement and portions of a node'sdedicated memory as well as the shared memory could be located off aremote SMC. In some implementations, the flit format of an existingprotocol can be reused and augmented to support a shared memoryinfrastructure. Indeed, a system may make use of the same foundationalflit format for point-to-point buffered memory links and shared memorylinks.

In one example, to accommodate the larger address space involved in ashared memory architecture, additional bits can be provided over andabove what is provided in the defined flit header format provided forGPIO and/or standard buffered memory access transactions. For instance,a slot of a multi-slot flit can be used to encode additional addressbits and extend the memory address scheme supported by the flit format(i.e., in SML transactions). For instance, FIG. 12 illustrates arepresentation 1200 showing an augmented version of the flit headerformat of an example high performance GPIO interconnect protocol, suchas the example format illustrated and described in connection with FIG.6. For a standard buffered memory link interface the payloads of Slots 0and 1 can be used to encoded the address of the line in memory that isto be accessed in the transaction (e.g., a 64 byte cache line address).In one implementation, additional address bits can be encoded in thebits of Slot 2 (i.e., in UIs 6, 7, and 8) as the additional, or upper,address bits of the pooled memory space. This can extend the addressspace by eight bits

Continuing with the example of FIG. 12, a three-slot flit format of theGPIO protocol introduced in FIG. 6 can be augmented with fields specificto transactions of a shared memory architecture including fields toassist in routing between multiple SMCs in this architecture. The upperaddress bits of the expanded memory address can also be used in lieu ofthe SMC number of the “Source Link ID” field, in some implementations,such as where a flat system address space exists where the upperphysical address bits represents the SMC number, among other examples.

As additional examples, a flit format of a GPIO protocol or bufferedmemory link protocol can be augmented to accommodate additional bits tooptionally indicate alternate route(s) to be taken if multiple pathsexist between a source-destination pair. For instance, in the example ofFIG. 12, reserved fields or a floating field provided in the header flitformat of a GPIO protocol can be encoded with ARR values for each of thetwo transactions (in Slots 0 and 1) to indicate a particular path totake between the SMCs of the shared memory architecture. For instance,ARR values can be utilized to provide high availability within thesystem. For instance, an SMC, or the node to which it is connected, candetect a time-out condition for a transaction. This can prompt a re-try(e.g., after one or more failed retries using the original routing path)using an available alternate path for routing the transaction betweenthe SMC and a destination SMC. To facilitate this, the SMC can encodebits of an ARR field for the transaction with a value to indicate thealternate path. This can allow a transaction to be retried along analternate path(s) to recover from the errors without escalating theerror to be unrecoverable (such as timeout or poison).

As noted above, a flit of a SML protocol can be based on, or be anaugmented version of, a flit with a base format defined by anotherinterconnect protocol (e.g., another protocol used in the system). Insome instances, the modifications to the base format can be selectable,in that the original, base flit/packet layout (of the other protocol)can be preserved in some cases, and the augmented version can be appliedin others. For instance, the flit can utilize one or more bits toindicate whether the flit is according to the base or modified flitformat. A receiving SMC (or node) can decode the flit to determine whichformat is applied to the flit. In one example, such as shown in FIG. 12,a bit can be provided to indicate whether Slot 2 of the base flit formatis being repurposes to provide extended memory address bits and/orsource node ID bits, among other examples. In one example, the bit canoccupy a defined virtual network bit in cases where only a singlevirtual network (e.g., VN0) is to be used. Indeed, the bit to indicatehow Slot 2 is to be used, can itself represent a modification to a baseflit format, among other examples. In one example, a Mode bit (e.g., inrow UI0 of the example of FIG. 12) can be reused to designate that Slot2 is to be used to extend the address is the base flit format, amongother examples.

As introduced above, in some implementations, a flit format can beprovided that indicates both an identifier of the source SMC and thedestination SMC. In some alternative embodiments, only the source SMCidentifier (or “node ID”) may be provided, and the destination SMC canbe identified implicitly from the address bits provided to indicate thecorresponding line of memory involved in the transaction. In otherwords, an SMC can determine that a particular line of memory referencedin the address field of a flit corresponds to a memory accessed througha particular SMC in a plurality of SMCs in a shared memory architecture.For instance, a record can be maintained at each SMC denoting whichaddresses corresponding to which memory elements connected directly towhich SMC. In other implementations, upper order address bits (e.g.,those added in the extension of a flit address field) can correspond toparticular SMCs within the shared memory architecture. Accordingly, thedestination SMC (connected to the target memory address) can beidentified by another SMC from the extended memory address field of theflit. The flits can be decoded by each SMC at each “hop” of the routingof a request flit to its destination. Each SMC can thus identify thedestination SMC from the address bits and determine the next hop toallow the request to progress toward the destination SMC.

Processor nodes in a shared memory architecture employing multipledifferent SMCs each managing a subset of the pooled and/or shared systemmemory, can be ignorant of the architecture used to access request lineof memory from its directly-connected (or “local”) SMC. For instance,the processor node can be blind to its local SMC determining thatanother SMC manages a requested portion of memory and forwarding thenode's request for handling by the other SMC. All the node sees is thatit sent a request to its local SMC and received a correspondingresponse. The node may also be ignorant to the fact that the requestedline of memory is in pooled memory. Consequently, in some cases, a nodemay send a flit corresponding to a request involving a particular lineof memory and send the flit according to a single-node buffered memorylink protocol (e.g., without the augmented fields provided for in ashared memory link protocol followed between SMCs). In such instances,at the ingress port of the local SMC, the additional enhancement fields(such as the ones shown in the examples of Table 1 or FIG. 12) can beattached to the flit (or packet) before sending it out within the SMC.On egress (e.g., to the processor node) these enhanced fields can bestripped away.

In some case, processor nodes may be ill equipped to properly tracktheir transactions, given the potential complexity involving multiplehops between SMCs in the shared memory architecture that may be hiddenfrom the processor nodes. Accordingly, a local SMC can tracktransactions on behalf of its respective processor node(s). The node'slocal SMC can preserve all the routing related fields that were includedin original flits of the request (e.g., as included in the flit asformatted according to the node's buffered memory link protocol (whichmay be different from the SML protocol)). As an SMC can then beresponsible for potentially large numbers of transactions, memory of theSMC itself can be allocated to accommodate the tracking of thetransaction information. In some instances, the SMC can optimize thenumber of bits tracked for each transaction by assigning or translatingfields of the original request to values with less bits than theoriginal. For instance, the SMC can assign an RHTID field value at anSMC-level with presumably less bits than the original. The transactioninformation can be stored in on-chip (i.e., on-SMC) storage with a CAMlook-up ability to retrieve the original bits on egress. The on-chipmemory can be supplemented (or replaced instead) with the local memory(e.g., buffered memory managed by the SMC) as storage above and beyondwhat would be available in on-chip memory, among other examples.

It should be appreciated that the implementations described herein areprovided as examples to illustrate certain principles and featuresdisclosed in the Specification. It should be appreciated thatalternative configurations, protocols, and architectures (other thanthose specifically discussed in the examples) can utilize and apply suchprinciples and features. As an example of one alternative, PCIe memoryread/write can be used (e.g., instead of SMI3 protocol) that is enhancedwith directory information. The directory information can be implementedthrough reserve bits of the PCIe packet. In another example, CPU nodescan utilize a cache controller (e.g., as an alternative to a sharedmemory controller) to send memory read/write transactions on a PCIelink, for instance, based on a remote address range check, among otherpotential examples and alternatives.

Turning to FIGS. 13A-13B, flowcharts 1300 a-b are shown illustratingexample techniques for communicating using a shared memory linkinterconnect. For instance, in FIG. 13A, a load/store memory accessmessage can be received 1305 at one of a plurality of shared memorycontrollers (SMCs) (e.g., over an SML link) from a processor node, themessage requesting data of a particular address of shared memory. Themessage can be embodied in one or more flits, including a header flit toidentify the particular address. The shared memory controller candetermine 1310 from the particular address (or a destination nodeidentifier identifying a particular one of the SMCs), that the message(or “request”) is to be handled by the particular SMC because theparticular SMC manages a particular one of the shared memory elements inthe system that hosts the particular address. In some cases, the SMCconnected directly to the processor node is the particular SMC. In othercases, the particular SMC is another one of the plurality of SMCs. TheSMC can determine whether it or another one of the SMCs is to handle therequest (e.g., at 1315). In cases where the SMC determines that anotherSMC is the particular SMC for handling the request, the SMC candetermine 1320 how to route the request to the particular SMC. In somecases this can involve sending the request to an intervening one of theplurality of SMCs. In other cases, the SMC can determine that therouting involves sending the request directly (e.g., over a single SMLlink) to the particular other SMC. The SMC then sends 1325 the request(e.g., over an SML link) to the other SMC (either the particular SMC orintervening SMC). In some implementations, the format of the flit usedby the processor node in its request may be augmented by the SMC tofacilitate routing of the flit within a network of SMCs. For instance,the SMC may utilize one of a plurality of slots to extend an addressindicated by the processor node, add a source or destination nodeidentifier to indicate the source or destination SMC in the transaction,encode bits to indicate that the particular flit slot is being used forthese purposes, among other modifications.

Once the particular SMC accesses the particular line of shared memorycorresponding to the particular address, the particular SMC can send aresponse back to the SMC connected to the particular node. The responsecan include enhanced fields (such as those included in the requestflits) that are used to assist in routing the response back to thesource SMC (e.g., the source node ID). After being routed back along thesame or a different path within a network of SMCs, the SMC connected tothe particular node can receive 1330 a response generated by theparticular SMC and can provide 1335 the response to the processor node.In cases where the flit format used between the SMCs represents anaugmented version of a flit consumed by the processor node, the SMC canstrip “extra” or enhanced fields from a response flit before providing1335 the response to the processor node. The response may appear to theprocessor node as having been handled entirely by the SMC it isconnected with. In other words, the node may be ignorant of that factthat the line of memory is managed by another SMC and that the requestwas routed over one or more other SMCs in a network of SMCs in a sharedmemory architecture.

In cases where the SMC determines (e.g., at 1315) that it manages thememory element hosting the requested line of memory, the SMC can access1340 the particular line of memory from the memory element, generate theresponse 1345, and provide 1335 the response to the processor node.

In the example of FIG. 13B, an SMC may receive 1350 a flit from anotherSMC in connection with the routing of the flit between a source SMC anda destination SMC. The source SMC is connected to a processor noderesponsible for generating the memory (e.g., load/store) requestinvolving a particular line of shared memory at a particular address.The destination SMC is connected to the memory element hosting theparticular line of shared memory. The destination SMC generates aresponse to the request upon accessing the particular line of sharedmemory. The flit can include fields such as an extended address field,source node ID to identify the source SMC, a destination node ID toidentify the destination SMC, an alternate routing path field toindicate a particular one of a plurality of available routing paths in anetwork of SMCs, among other fields. Such fields can be included in bothrequest and response flits. Further, the fields can be added to augmenta base flit format, such as a flit format defined according to a singlenode buffered memory access protocol, a GPIO protocol, or anotherprotocol. Each SMC in the routing path can decode 1355 the flit todetermine 1360 the next hop in the routing. For instance, in cases wherethe flit represents a request, the flit can be decoded to identify theextended address of the particular line of memory. The SMC can determinefrom the extended address the destination SMC and determine how (e.g.,either directly or over which intermediate SMC) to forward the requestto the destination SMC. In other instances, the SMC can alternatively(or additionally) consult a destination node ID field of the flit. Incases where the flit embodies at least a portion of a response to arequest (generated by the destination SMC), the SMC can decode 1355 theflit to identify a source node ID corresponding to the source SMC anddetermine a next step in a corresponding routing path to route theresponse flit to the source SMC for delivery to the processor nodeoriginating the transaction. In some cases, this return routing path maybe the identical (but reverse) path used to send the request from thesource SMC to the destination SMC. In other cases, an alternate path canbe used to send the response from the source SMC to the destination SMC.

It should be noted that while much of the above principles and examplesare described within the context of PCIe and particular revisions of thePCIe specification, the principles, solutions, and features describedherein can be equally applicable to other protocols and systems. Forinstance, analogous lane errors can be detected in other links usingother protocols based on analogous symbols, data streams, and tokens, aswell as rules specified for the use, placement, and formatting of suchstructures within data transmitted over these other links. Further,alternative mechanisms and structures (e.g., beside a PCIe LES registeror SKP OS) can be used to provide lane error detection and reportingfunctionality within a system. Moreover, combinations of the abovesolutions can be applied within systems, including combinations oflogical and physical enhancements to a link and its corresponding logicas described herein, among other examples.

Note that the apparatus', methods', and systems described above may beimplemented in any electronic device or system as aforementioned. Asspecific illustrations, the figures below provide exemplary systems forutilizing the invention as described herein. As the systems below aredescribed in more detail, a number of different interconnects aredisclosed, described, and revisited from the discussion above. And as isreadily apparent, the advances described above may be applied to any ofthose interconnects, fabrics, or architectures.

Referring to FIG. 14, an embodiment of a block diagram for a computingsystem including a multicore processor is depicted. Processor 1400includes any processor or processing device, such as a microprocessor,an embedded processor, a digital signal processor (DSP), a networkprocessor, a handheld processor, an application processor, aco-processor, a system on a chip (SOC), or other device to execute code.Processor 1400, in one embodiment, includes at least two cores—core 1401and 1402, which may include asymmetric cores or symmetric cores (theillustrated embodiment). However, processor 1400 may include any numberof processing elements that may be symmetric or asymmetric.

In one embodiment, a processing element refers to hardware or logic tosupport a software thread. Examples of hardware processing elementsinclude: a thread unit, a thread slot, a thread, a process unit, acontext, a context unit, a logical processor, a hardware thread, a core,and/or any other element, which is capable of holding a state for aprocessor, such as an execution state or architectural state. In otherwords, a processing element, in one embodiment, refers to any hardwarecapable of being independently associated with code, such as a softwarethread, operating system, application, or other code. A physicalprocessor (or processor socket) typically refers to an integratedcircuit, which potentially includes any number of other processingelements, such as cores or hardware threads.

A core often refers to logic located on an integrated circuit capable ofmaintaining an independent architectural state, wherein eachindependently maintained architectural state is associated with at leastsome dedicated execution resources. In contrast to cores, a hardwarethread typically refers to any logic located on an integrated circuitcapable of maintaining an independent architectural state, wherein theindependently maintained architectural states share access to executionresources. As can be seen, when certain resources are shared and othersare dedicated to an architectural state, the line between thenomenclature of a hardware thread and core overlaps. Yet often, a coreand a hardware thread are viewed by an operating system as individuallogical processors, where the operating system is able to individuallyschedule operations on each logical processor.

Physical processor 1400, as illustrated in FIG. 14, includes twocores—core 1401 and 1402. Here, core 1401 and 1402 are consideredsymmetric cores, i.e. cores with the same configurations, functionalunits, and/or logic. In another embodiment, core 1401 includes anout-of-order processor core, while core 1402 includes an in-orderprocessor core. However, cores 1401 and 1402 may be individuallyselected from any type of core, such as a native core, a softwaremanaged core, a core adapted to execute a native Instruction SetArchitecture (ISA), a core adapted to execute a translated InstructionSet Architecture (ISA), a co-designed core, or other known core. In aheterogeneous core environment (i.e. asymmetric cores), some form oftranslation, such a binary translation, may be utilized to schedule orexecute code on one or both cores. Yet to further the discussion, thefunctional units illustrated in core 1401 are described in furtherdetail below, as the units in core 1402 operate in a similar manner inthe depicted embodiment.

As depicted, core 1401 includes two hardware threads 1401 a and 1401 b,which may also be referred to as hardware thread slots 1401 a and 1401b. Therefore, software entities, such as an operating system, in oneembodiment potentially view processor 1400 as four separate processors,i.e., four logical processors or processing elements capable ofexecuting four software threads concurrently. As alluded to above, afirst thread is associated with architecture state registers 1401 a, asecond thread is associated with architecture state registers 1401 b, athird thread may be associated with architecture state registers 1402 a,and a fourth thread may be associated with architecture state registers1402 b. Here, each of the architecture state registers (1401 a, 1401 b,1402 a, and 1402 b) may be referred to as processing elements, threadslots, or thread units, as described above. As illustrated, architecturestate registers 1401 a are replicated in architecture state registers1401 b, so individual architecture states/contexts are capable of beingstored for logical processor 1401 a and logical processor 1401 b. Incore 1401, other smaller resources, such as instruction pointers andrenaming logic in allocator and renamer block 1430 may also bereplicated for threads 1401 a and 1401 b. Some resources, such asre-order buffers in reorder/retirement unit 1435, ILTB 1420, load/storebuffers, and queues may be shared through partitioning. Other resources,such as general purpose internal registers, page-table base register(s),low-level data-cache and data-TLB 1415, execution unit(s) 1440, andportions of out-of-order unit 1435 are potentially fully shared.

Processor 1400 often includes other resources, which may be fullyshared, shared through partitioning, or dedicated by/to processingelements. In FIG. 14, an embodiment of a purely exemplary processor withillustrative logical units/resources of a processor is illustrated. Notethat a processor may include, or omit, any of these functional units, aswell as include any other known functional units, logic, or firmware notdepicted. As illustrated, core 1401 includes a simplified,representative out-of-order (OOO) processor core. But an in-orderprocessor may be utilized in different embodiments. The OOO coreincludes a branch target buffer 1420 to predict branches to beexecuted/taken and an instruction-translation buffer (I-TLB) 1420 tostore address translation entries for instructions.

Core 1401 further includes decode module 1425 coupled to fetch unit 1420to decode fetched elements. Fetch logic, in one embodiment, includesindividual sequencers associated with thread slots 1401 a, 1401 b,respectively. Usually core 1401 is associated with a first ISA, whichdefines/specifies instructions executable on processor 1400. Oftenmachine code instructions that are part of the first ISA include aportion of the instruction (referred to as an opcode), whichreferences/specifies an instruction or operation to be performed. Decodelogic 1425 includes circuitry that recognizes these instructions fromtheir opcodes and passes the decoded instructions on in the pipeline forprocessing as defined by the first ISA. For example, as discussed inmore detail below decoders 1425, in one embodiment, include logicdesigned or adapted to recognize specific instructions, such astransactional instruction. As a result of the recognition by decoders1425, the architecture or core 1401 takes specific, predefined actionsto perform tasks associated with the appropriate instruction. It isimportant to note that any of the tasks, blocks, operations, and methodsdescribed herein may be performed in response to a single or multipleinstructions; some of which may be new or old instructions. Notedecoders 1426, in one embodiment, recognize the same ISA (or a subsetthereof). Alternatively, in a heterogeneous core environment, decoders1426 recognize a second ISA (either a subset of the first ISA or adistinct ISA).

In one example, allocator and renamer block 1430 includes an allocatorto reserve resources, such as register files to store instructionprocessing results. However, threads 1401 a and 1401 b are potentiallycapable of out-of-order execution, where allocator and renamer block1430 also reserves other resources, such as reorder buffers to trackinstruction results. Unit 1430 may also include a register renamer torename program/instruction reference registers to other registersinternal to processor 1400. Reorder/retirement unit 1435 includescomponents, such as the reorder buffers mentioned above, load buffers,and store buffers, to support out-of-order execution and later in-orderretirement of instructions executed out-of-order.

Scheduler and execution unit(s) block 1440, in one embodiment, includesa scheduler unit to schedule instructions/operation on execution units.For example, a floating point instruction is scheduled on a port of anexecution unit that has an available floating point execution unit.Register files associated with the execution units are also included tostore information instruction processing results. Exemplary executionunits include a floating point execution unit, an integer executionunit, a jump execution unit, a load execution unit, a store executionunit, and other known execution units.

Lower level data cache and data translation buffer (D-TLB) 1450 arecoupled to execution unit(s) 1440. The data cache is to store recentlyused/operated on elements, such as data operands, which are potentiallyheld in memory coherency states. The D-TLB is to store recentvirtual/linear to physical address translations. As a specific example,a processor may include a page table structure to break physical memoryinto a plurality of virtual pages.

Here, cores 1401 and 1402 share access to higher-level or further-outcache, such as a second level cache associated with on-chip interface1410. Note that higher-level or further-out refers to cache levelsincreasing or getting further way from the execution unit(s). In oneembodiment, higher-level cache is a last-level data cache—last cache inthe memory hierarchy on processor 1400—such as a second or third leveldata cache. However, higher level cache is not so limited, as it may beassociated with or include an instruction cache. A trace cache—a type ofinstruction cache—instead may be coupled after decoder 1425 to storerecently decoded traces. Here, an instruction potentially refers to amacro-instruction (i.e. a general instruction recognized by thedecoders), which may decode into a number of micro-instructions(micro-operations).

In the depicted configuration, processor 1400 also includes on-chipinterface module 1410. Historically, a memory controller, which isdescribed in more detail below, has been included in a computing systemexternal to processor 1400. In this scenario, on-chip interface 1410 isto communicate with devices external to processor 1400, such as systemmemory 1475, a chipset (in some cases including a memory controller hubor shared memory controller to connect to memory 1475 and an I/Ocontroller hub to connect peripheral devices), a memory controller hub,a northbridge, or other integrated circuit. And in this scenario, link1405 may include any known interconnect, such as multi-drop bus, apoint-to-point interconnect, a serial interconnect, a parallel bus, acoherent (e.g. cache coherent) bus, a layered protocol architecture, adifferential bus, a GTL bus, or an SML link.

Memory 1475 may be dedicated to processor 1400 or shared with otherdevices in a system. Common examples of types of memory 1475 includeDRAM, SRAM, non-volatile memory (NV memory), and other known storagedevices. Note that device 1480 may include a graphic accelerator,processor or card coupled to a memory controller hub, data storagecoupled to an I/O controller hub, a wireless transceiver, a flashdevice, an audio controller, a network controller, or other knowndevice.

Recently however, as more logic and devices are being integrated on asingle die, such as SOC, each of these devices may be incorporated onprocessor 1400. For example in one embodiment, a memory controller hubis on the same package and/or die with processor 1400. Here, a portionof the core (an on-core portion) 1410 includes one or more controller(s)for interfacing with other devices such as memory 1475 or a graphicsdevice 1480. The configuration including an interconnect and controllersfor interfacing with such devices is often referred to as an on-core (orun-core configuration). As an example, on-chip interface 1410 includes aring interconnect for on-chip communication and a high-speed serialpoint-to-point link 1405 for off-chip communication. Yet, in the SOCenvironment, even more devices, such as the network interface,co-processors, memory 1475, graphics processor 1480, and any other knowncomputer devices/interface may be integrated on a single die orintegrated circuit to provide small form factor with high functionalityand low power consumption.

In one embodiment, processor 1400 is capable of executing a compiler,optimization, and/or translator code 1477 to compile, translate, and/oroptimize application code 1476 to support the apparatus and methodsdescribed herein or to interface therewith. A compiler often includes aprogram or set of programs to translate source text/code into targettext/code. Usually, compilation of program/application code with acompiler is done in multiple phases and passes to transform hi-levelprogramming language code into low-level machine or assembly languagecode. Yet, single pass compilers may still be utilized for simplecompilation. A compiler may utilize any known compilation techniques andperform any known compiler operations, such as lexical analysis,preprocessing, parsing, semantic analysis, code generation, codetransformation, and code optimization.

Larger compilers often include multiple phases, but most often thesephases are included within two general phases: (1) a front-end, i.e.generally where syntactic processing, semantic processing, and sometransformation/optimization may take place, and (2) a back-end, i.e.generally where analysis, transformations, optimizations, and codegeneration takes place. Some compilers refer to a middle, whichillustrates the blurring of delineation between a front-end and back endof a compiler. As a result, reference to insertion, association,generation, or other operation of a compiler may take place in any ofthe aforementioned phases or passes, as well as any other known phasesor passes of a compiler. As an illustrative example, a compilerpotentially inserts operations, calls, functions, etc. in one or morephases of compilation, such as insertion of calls/operations in afront-end phase of compilation and then transformation of thecalls/operations into lower-level code during a transformation phase.Note that during dynamic compilation, compiler code or dynamicoptimization code may insert such operations/calls, as well as optimizethe code for execution during runtime. As a specific illustrativeexample, binary code (already compiled code) may be dynamicallyoptimized during runtime. Here, the program code may include the dynamicoptimization code, the binary code, or a combination thereof.

Similar to a compiler, a translator, such as a binary translator,translates code either statically or dynamically to optimize and/ortranslate code. Therefore, reference to execution of code, applicationcode, program code, or other software environment may refer to: (1)execution of a compiler program(s), optimization code optimizer, ortranslator either dynamically or statically, to compile program code, tomaintain software structures, to perform other operations, to optimizecode, or to translate code; (2) execution of main program code includingoperations/calls, such as application code that has beenoptimized/compiled; (3) execution of other program code, such aslibraries, associated with the main program code to maintain softwarestructures, to perform other software related operations, or to optimizecode; or (4) a combination thereof.

While the present invention has been described with respect to a limitednumber of embodiments, those skilled in the art will appreciate numerousmodifications and variations therefrom. It is intended that the appendedclaims cover all such modifications and variations as fall within thetrue spirit and scope of this present invention.

A design may go through various stages, from creation to simulation tofabrication. Data representing a design may represent the design in anumber of manners. First, as is useful in simulations, the hardware maybe represented using a hardware description language or anotherfunctional description language. Additionally, a circuit level modelwith logic and/or transistor gates may be produced at some stages of thedesign process. Furthermore, most designs, at some stage, reach a levelof data representing the physical placement of various devices in thehardware model. In the case where conventional semiconductor fabricationtechniques are used, the data representing the hardware model may be thedata specifying the presence or absence of various features on differentmask layers for masks used to produce the integrated circuit. In anyrepresentation of the design, the data may be stored in any form of amachine readable medium. A memory or a magnetic or optical storage suchas a disc may be the machine readable medium to store informationtransmitted via optical or electrical wave modulated or otherwisegenerated to transmit such information. When an electrical carrier waveindicating or carrying the code or design is transmitted, to the extentthat copying, buffering, or re-transmission of the electrical signal isperformed, a new copy is made. Thus, a communication provider or anetwork provider may store on a tangible, machine-readable medium, atleast temporarily, an article, such as information encoded into acarrier wave, embodying techniques of embodiments of the presentinvention.

A module as used herein refers to any combination of hardware, software,and/or firmware. As an example, a module includes hardware, such as amicro-controller, associated with a non-transitory medium to store codeadapted to be executed by the micro-controller. Therefore, reference toa module, in one embodiment, refers to the hardware, which isspecifically configured to recognize and/or execute the code to be heldon a non-transitory medium. Furthermore, in another embodiment, use of amodule refers to the non-transitory medium including the code, which isspecifically adapted to be executed by the microcontroller to performpredetermined operations. And as can be inferred, in yet anotherembodiment, the term module (in this example) may refer to thecombination of the microcontroller and the non-transitory medium. Oftenmodule boundaries that are illustrated as separate commonly vary andpotentially overlap. For example, a first and a second module may sharehardware, software, firmware, or a combination thereof, whilepotentially retaining some independent hardware, software, or firmware.In one embodiment, use of the term logic includes hardware, such astransistors, registers, or other hardware, such as programmable logicdevices.

Use of the phrase ‘configured to,’ in one embodiment, refers toarranging, putting together, manufacturing, offering to sell, importingand/or designing an apparatus, hardware, logic, or element to perform adesignated or determined task. In this example, an apparatus or elementthereof that is not operating is still ‘configured to’ perform adesignated task if it is designed, coupled, and/or interconnected toperform said designated task. As a purely illustrative example, a logicgate may provide a 0 or a 1 during operation. But a logic gate‘configured to’ provide an enable signal to a clock does not includeevery potential logic gate that may provide a 1 or 0. Instead, the logicgate is one coupled in some manner that during operation the 1 or 0output is to enable the clock. Note once again that use of the term‘configured to’ does not require operation, but instead focus on thelatent state of an apparatus, hardware, and/or element, where in thelatent state the apparatus, hardware, and/or element is designed toperform a particular task when the apparatus, hardware, and/or elementis operating.

Furthermore, use of the phrases ‘to,’ capable of/to,′ and or ‘operableto,’ in one embodiment, refers to some apparatus, logic, hardware,and/or element designed in such a way to enable use of the apparatus,logic, hardware, and/or element in a specified manner. Note as abovethat use of to, capable to, or operable to, in one embodiment, refers tothe latent state of an apparatus, logic, hardware, and/or element, wherethe apparatus, logic, hardware, and/or element is not operating but isdesigned in such a manner to enable use of an apparatus in a specifiedmanner.

A value, as used herein, includes any known representation of a number,a state, a logical state, or a binary logical state. Often, the use oflogic levels, logic values, or logical values is also referred to as 1'sand 0's, which simply represents binary logic states. For example, a 1refers to a high logic level and 0 refers to a low logic level. In oneembodiment, a storage cell, such as a transistor or flash cell, may becapable of holding a single logical value or multiple logical values.However, other representations of values in computer systems have beenused. For example the decimal number ten may also be represented as abinary value of 1410 and a hexadecimal letter A. Therefore, a valueincludes any representation of information capable of being held in acomputer system.

Moreover, states may be represented by values or portions of values. Asan example, a first value, such as a logical one, may represent adefault or initial state, while a second value, such as a logical zero,may represent a non-default state. In addition, the terms reset and set,in one embodiment, refer to a default and an updated value or state,respectively. For example, a default value potentially includes a highlogical value, i.e. reset, while an updated value potentially includes alow logical value, i.e. set. Note that any combination of values may beutilized to represent any number of states.

The embodiments of methods, hardware, software, firmware or code setforth above may be implemented via instructions or code stored on amachine-accessible, machine readable, computer accessible, or computerreadable medium which are executable by a processing element. Anon-transitory machine-accessible/readable medium includes any mechanismthat provides (i.e., stores and/or transmits) information in a formreadable by a machine, such as a computer or electronic system. Forexample, a non-transitory machine-accessible medium includesrandom-access memory (RAM), such as static RAM (SRAM) or dynamic RAM(DRAM); ROM; magnetic or optical storage medium; flash memory devices;electrical storage devices; optical storage devices; acoustical storagedevices; other form of storage devices for holding information receivedfrom transitory (propagated) signals (e.g., carrier waves, infraredsignals, digital signals); etc, which are to be distinguished from thenon-transitory mediums that may receive information there from.

Instructions used to program logic to perform embodiments of theinvention may be stored within a memory in the system, such as DRAM,cache, flash memory, or other storage. Furthermore, the instructions canbe distributed via a network or by way of other computer readable media.Thus a machine-readable medium may include any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer), but is not limited to, floppy diskettes, optical disks,Compact Disc, Read-Only Memory (CD-ROMs), and magneto-optical disks,Read-Only Memory (ROMs), Random Access Memory (RAM), ErasableProgrammable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), magnetic or optical cards, flashmemory, or a tangible, machine-readable storage used in the transmissionof information over the Internet via electrical, optical, acoustical orother forms of propagated signals (e.g., carrier waves, infraredsignals, digital signals, etc.). Accordingly, the computer-readablemedium includes any type of tangible machine-readable medium suitablefor storing or transmitting electronic instructions or information in aform readable by a machine (e.g., a computer).

The following examples pertain to embodiments in accordance with thisSpecification. One or more embodiments may provide an apparatus, asystem, a machine readable storage, a machine readable medium, a method,and hardware- and/or software-based logic (e.g., implemented in a sharedmemory controller) to a first interface to receive a flit associatedwith a memory transaction. The particular shared memory controller is tocontrol access to a portion of a shared memory resource having aplurality of memory elements. The flit can include a node identifierfield to identify a particular one of a plurality of processor nodes asa source of the transaction.

In one example, the flit includes at least three slots.

In one example, at least two of the slots are to identify addresses oftwo different transactions and a third of the at least three slots is toprovide bits to extend the addresses identified in the at least two ofthe slots.

In one example, the flit is to further include a field to indicatewhether the third slot is used to extend the addresses identified in theat least two of the slots.

In one example, the flit has a format of a single node buffered memorylink protocol when the filed indicates that the third slot is not usedto extend ranges of the addresses identified in the at least two of theslots.

In one example, the particular shared memory controller is further todecode the flit and determine a routing for the flit based on thedecoding.

In one example, the particular shared memory controller further includesa second interface to route the flit to another one of the plurality ofshared memory controllers based on the decoding.

In one example, the other shared memory controller is determinedimplicitly from an address field of the flit.

In one example, the other shared memory controller is determined from ashared memory controller number encoded in a destination node field ofthe flit.

In one example, the particular shared memory controller directlyconnects to and manages access to a particular subset of the pluralityof memory elements corresponding to the portion of the shared memoryresource.

In one example, the first interface includes a layered protocol stack.

In one example, the first interface utilizes a shared memory linkprotocol, and the shared memory link protocol utilizes physical layerlogic of a different interconnect protocol.

In one example, the different interconnect protocol includes aPeripheral Component Interconnect Express (PCIe)-based protocol.

One or more embodiments may provide an apparatus, a system, a machinereadable storage, a machine readable medium, a method, and hardware-and/or software-based logic (e.g., implemented in a shared memorycontroller) to receive a flit from another first shared memorycontroller over a shared memory link, where the flit includes a nodeidentifier (ID) field and an address of a particular line of the sharedmemory, and the node ID field identifies that the first shared memorycontroller corresponds to a source of the flit. Further, a second sharedmemory controller is determined from at least the address field of theflit, where the second shared memory controller is connected to a memoryelement corresponding to the particular line. The flit can be forwardedto the second shared memory controller using a shared memory linkaccording to a routing path.

In one example, the routing path to the second shared memory controlleris to be determined.

In one example, the routing path is one of a plurality of alternaterouting paths to the second shared memory controller and the sharedmemory controller is to determine a particular one of the plurality ofalternate routing paths from a field in the flit.

In one example, another flit is generated to correspond to anothertransaction to involve another line of the shared memory, the other flitis to include a node ID field and an address of the other line of theshared memory, and the node ID field of the other flit is to identifythat the shared memory controller corresponds to a source of the othertransaction.

One or more embodiments may provide a system that includes a sharedmemory including a plurality of memory elements, a shared memorycontroller to manage access to a subset of the plurality of memoryelements, and at least one processor node connected to the shared memorycontroller by a shared memory link. The processor node is to generate aflit to correspond to a transaction to involve a particular address ofthe shared memory and send the flit to the shared memory controller. Theshared memory controller can further identify a particular one of aplurality of shared memory controllers that corresponds to theparticular address and determine a routing path for providing the flitto the particular shared memory.

In one example, the shared memory controller is to deliver a result ofthe transaction to the processor node.

In one example, the shared memory controller includes a first sharedmemory controller, the subset of the plurality of memory elementsincludes a first subset of the memory elements. The processor nodeincludes a first processor node, and the system further includes a firstdevice including the first shared memory controller, the first subset ofthe memory elements, and the one or more nodes, and a second deviceconnected to the first device by a shared memory link, where the seconddevice includes a second shared memory controller, a second subset ofthe memory elements, and a second processor node.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

In the foregoing specification, a detailed description has been givenwith reference to specific exemplary embodiments. It will, however, beevident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense. Furthermore, the foregoing use of embodiment andother exemplarily language does not necessarily refer to the sameembodiment or the same example, but may refer to different and distinctembodiments, as well as potentially the same embodiment.

What is claimed is:
 1. An apparatus comprising: a first device comprising at least a portion of an interconnect fabric, a first shared memory controller to control access to first shared memory, and a first plurality of processor nodes, wherein the first device further comprises an interface to receive a data request from a second device over a data link, the interface comprises physical layer circuitry to implement a physical layer for the data link according to a physical layer protocol, and the interface further comprises first link layer circuitry to implement a first link layer protocol and second link layer circuitry to implement a different second link layer protocol, wherein the data link comprises a plurality of lanes, the data link supports multiplexing of data of the different first and second link layer protocols over the same physical layer of the data link, the first shared memory is not directly connected to the processor nodes, and the second device comprises a second shared memory controller to couple and control access to second shared memory, and a second plurality of processor nodes, wherein the data request comprises link layer data, the data request is addressed to be routed to the first memory controller through the interconnect fabric from the second device, and the first memory controller is to handle the data request, wherein the data request comprises a node identifier to identify a particular processor node of the second plurality of processor nodes as a source of the data request, and the data request further comprises an address to identify a portion of the first shared memory associated with the data request.
 2. The apparatus of claim 1, wherein the first and second shared memory comprise shared memory of a platform comprising the first device and the second device.
 3. The apparatus of claim 1, wherein the first shared memory comprises dynamic random-access memory (DRAM).
 4. The apparatus of claim 3, wherein the first shared memory comprises dual in-line memory module (DIMM) memory.
 5. The apparatus of claim 1, wherein the first memory controller is coupled directly to the first shared memory by a double data rate (DDR) connection.
 6. The apparatus of claim 1, wherein the data link comprises a physical layer based at least in part on a Peripheral Component Interconnect Express (PCIe)-based physical layer.
 7. The apparatus of claim 1, wherein the interface comprises a serializer/deserializer (SerDes).
 8. The apparatus of claim 1, wherein the data link comprises a multilane link.
 9. The apparatus of claim 8, wherein the multilane link comprises at least eight lanes.
 10. The apparatus of claim 1, wherein the data request comprises one of a read request or write request of the portion of the first shared memory.
 11. A system comprising: a first die comprising at least a portion of an interconnect fabric, a first memory controller to couple to first shared memory, and a first plurality of processor nodes, wherein the first plurality of processor nodes are not directly connected; and a second die, coupled to the first die by a data link, wherein the second die comprises a second memory controller to couple to second shared memory, and a second plurality of processor nodes, wherein the data link comprises a plurality of lanes, the data link supports multiplexing of data of different link layer protocols over a common physical layer of the data link, a particular one of the first plurality of processor nodes is to send a memory request, the memory request is to be routed to the second memory controller, and the second memory controller is to handle the memory request, wherein the memory request comprises link layer data comprising a node identifier to identify the particular processor node and an address to identify a portion of the second shared memory.
 12. The system of claim 11, wherein the second die implements at least a portion of the interconnect fabric.
 13. The system of claim 11, wherein the system comprises a rack server system.
 14. The system of claim 11, wherein the first and second memory comprise shared memory of the system.
 15. The system of claim 11, wherein the first and second memory each respectively comprise dynamic random-access memory (DRAM).
 16. The system of claim 15, wherein the first and second memory each respectively comprise dual in-line memory module (DIMM) memory. 